Electronic device and method of operating the same

ABSTRACT

An electronic device and a method of operating the electronic device are provided. The method includes obtaining a first image including an interesting object and at least one peripheral object, obtaining relevance of the at least one peripheral object with respect to the interesting object based on at least one of attributes including a state, a motion, and a location of the interesting object, and replacing an area of the whole area of the first image, which is occupied by the interesting object and the other remaining peripheral objects than a specific peripheral object determined based on the relevance, with a second image. According to the embodiments of the present invention, the electronic device and method may provide a vivid video conference.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Application No.61/541,289, filed on Sep. 30, 2011, the contents of which are hereinincorporated by reference in its entirety.

BACKGROUND

1. Technical Field

The embodiments of the present invention are directed to an electronicdevice and a method of operating the electronic device, and morespecifically to an electronic device that may be used for avideoconference and a method of operating the electronic device.

2. Discussion of the Related Art

Tele-presence refers to a set of technologies which allow a person tofeel as if they were present. Tele-presence technologies reproduceinformation on five senses a person feels in a specific space at aremote location. Element technologies for tele-presence may includevideo, audio, tactile, and network transmission technologies. Suchtele-presence technologies are adopted for video conference systems.Tele-presence-based video conference systems provide higher-qualitycommunications and allow users to further concentrate on theconversation compared to conventional video conference systems.

The tele-presence technologies for teleconference systems, althoughshowing a little difference for each and every manufacturer, may beapplicable to video, audio, and network transmission technologies asfollows:

For video technologies, the tele-presence technologies apply asgenerating natural eye-contact images for being able to make a userfurther feel like he would face another user and generatinghigh-resolution images. For audio technologies, the tele-presencetechnologies apply as audio playback technologies that may create afeeling of a space based on a speaker's location. For networktransmission technologies, the tele-presence technologies apply asreal-time image/sound transmission technologies based on an MCU (MultiControl Unit).

In contrast to video, audio, and network transmission for videoconference systems which have been actively researched, data sharingbetween attendants in a conference is still not satisfactory. Currentvideo conference systems use a separate monitor for data sharing.Accordingly, when a user shifts his eyes from an image screen to a datascreen, the eye contact is not maintained lowering a feeling as ifactually facing another user. Moreover, a short drop in conversationoccurs at every data manipulation because the data manipulation isconducted by a peripheral device, such as a mouse.

SUMMARY

The embodiments of the present invention provide an electronic deviceand a method of operating the electronic device, which may allow for avivid teleconference.

According to an embodiment, there is provided a method of operating anelectronic device, the method including obtaining a first imageincluding an interesting object and at least one peripheral object,obtaining relevance of the at least one peripheral object with respectto the interesting object based on at least one of attributes includinga state, a motion, and a location of the interesting object, andreplacing an area of the whole area of the first image, which isoccupied by the interesting object and the other remaining peripheralobjects than a specific peripheral object determined based on therelevance, with a second image.

Obtaining the relevance of the at least one peripheral object isperformed in consideration of whether part of the interesting object isconnected to the peripheral object.

Obtaining the relevance of the at least one peripheral object isperformed in consideration of a motion direction in which at least partof the interesting object moves.

Obtaining the relevance of the at least one peripheral object isperformed in consideration of a distance between the interesting objectand the at least one peripheral object.

Obtaining the relevance of the at least one peripheral object isperformed in further consideration of a user's input on the at least oneperipheral object.

The method further includes displaying the first image and receiving theuser's input on the displayed first image.

Displaying the first image includes making the interesting object andthe first peripheral object different in display property from theremaining peripheral objects.

The method further includes transmitting a third image generated byreplacing the area occupied by the remaining peripheral objects with thesecond image to a second electronic device.

The first image includes a moving image.

According to an embodiment, there is provided a method of operating anelectronic device, the method including obtaining a first image of aninteresting object, obtaining a virtual image corresponding to aspecific object determined based on a first motion of the interestingobject, obtaining a final image including the virtual image and at leastpart of the first image in consideration of a display state of thevirtual image determined based on a second object of the second motion,and transmitting the final image to a second electronic device.

The display state of the virtual image includes at least one of alocation where the virtual image is to be synthesized, a size, anorientation, a rotational state, and a marking state of the virtualimage.

Obtaining the final image includes sensing the second motion in realtime and changing the display state of the virtual image based on thesensed second motion.

Obtaining the virtual image includes determining whether the firstmotion is associated with the specific object and selecting the specificobject only when the first motion is associated with the specificobject.

The virtual image is a 2D image or a 3D image.

According to the embodiments of the present invention, the followingeffects may be achieved.

First, a user who hosts a conference and/or a distance lecture throughan electronic device according to an embodiment of the present inventionmay transmit an image including only his desired objects (e.g., his andspecific peripheral objects) in the place where he hosts the conferenceto an electronic device that another user uses. In particular, since aspecific object desired to be together displayed may be selected withouta complicated procedure, the distance conference may be smoothlyperformed.

Second, a user who hosts a teleconference and/or a distance lecture neednot previously create digitalized data of materials, such as audiovisualdocumentation, necessary for the conference and share the data withattendants.

Third, the embodiments may provide effects of being able to allow theobjects, such as handwritten data or shape of a prototype model, to beimmediately shared with other attendants while the conference is on theway. For example, a virtual image for an actual object may be instantlyobtained and synthesized with an image for the teleconference, and thesynthesized image may be transmitted to other users, thereby enablingdata necessary for the conference to be immediately shared with theusers in a convenience way.

Fourth, since data (e.g., objects) to be shared is stored as virtualimages and synthesized with images to be transmitted, a user hosting theconference may easily share desired objects with other users without aneffort to make to-be-shared objects oriented toward the camera (forexample, an effort for the host to proceed with the conference whileholding the to-be-shared data).

Finally, a user may control the displayed location, size, orientation,and a display state, such as rotational state or marking state, of theinteresting object to be shared based on his gesture, so that theteleconference and/or distance lecture may be performed more smoothly.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the present invention will become readily apparent byreference to the following detailed description when considered inconjunction with the accompanying drawings wherein:

FIG. 1 is a block diagram illustrating an electronic device according toan embodiment of the present invention;

FIG. 2 is a view illustrating an example where a user inputs a gestureto an electronic device as shown in FIG. 1.

FIG. 3 is a view for describing an environment according to anembodiment of the present invention;

FIG. 4 is a flowchart for describing a method of operating an electronicdevice according to an embodiment of the present invention;

FIG. 5 is a view for describing a first image obtained according to anembodiment of the present invention;

FIGS. 6 to 10 are views for describing a method of obtaining relevanceof a peripheral object for an interesting object based on an attributeof the interesting object according to an embodiment of the presentinvention;

FIGS. 11 and 12 are views each illustrating a second image in which anarea occupied by the remaining objects is replaced with another imageaccording to an embodiment of the present invention;

FIGS. 13 and 14 are views for describing a method of selecting aspecific object according to an embodiment of the present invention;

FIG. 15 is a flowchart for describing a method of operating anelectronic device according to an embodiment of the present invention;

FIG. 16 is a view for describing a first place according to anembodiment of the present invention;

FIGS. 17 to 23 are views for describing the type of first motions and amethod of determining specific objects based on the first motionsaccording to an embodiment of the present invention;

FIGS. 24 and 25 are views for describing a method of changing a displaystatus of a virtual image according to an embodiment of the presentinvention; and

FIGS. 26A, 26B, and 26C are views for describing a rotational status ofa virtual image according to an embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

The present invention will now be described more fully with reference tothe accompanying drawings, in which exemplary embodiments of theinvention are shown. The invention may, however, be embodied in manydifferent forms and should not be construed as being limited to theembodiments set forth herein; rather, there embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the concept of the invention to those skilled in the art.

Hereinafter, a mobile terminal relating to the present invention will bedescribed below in more detail with reference to the accompanyingdrawings. In the following description, suffixes “module” and “unit” aregiven to components of the mobile terminal in consideration of onlyfacilitation of description and do not have meanings or functionsdiscriminated from each other.

FIG. 1 is a block diagram illustrating an electronic device according toan embodiment of the present invention.

Referring to FIG. 1, the electronic device 100 includes a communicationunit 110, a user input unit 120, an output unit 150, a memory 160, aninterface unit 170, a control unit 180, and a power supply unit 190. Thecomponents shown in FIG. 1 may be components that may be commonlyincluded in an electronic device. Accordingly, more or less componentsmay be included in the electronic device 100.

The communication unit 110 may include one or more modules that enablecommunication between the electronic device 100 and a communicationsystem or between the electronic device 100 and another device. Forinstance, the communication unit 110 may include a broadcast receivingunit 111, an Internet module 113, and a near-field communication module114.

The broadcast receiving unit 111 receives broadcast signals and/orbroadcast-related information from an external broadcast managing serverthrough a broadcast channel.

The broadcast channel may include a satellite channel and a terrestrialchannel. The broadcast managing server may refer to a server thatgenerates broadcast signals and/or broadcast-related information andbroadcasts the signals and/or information or a server that receivespre-generated broadcast signals and/or broadcast-related information andbroadcasts the signals and/or information to a terminal. The broadcastsignals may include TV broadcast signals, radio broadcast signals, databroadcast signals as well as combinations of TV broadcast signals orradio broadcast signals and data broadcast signals.

The broadcast-related information may refer to information relating tobroadcast channels, broadcast programs, or broadcast service providers.The broadcast-related information may be provided through acommunication network.

The broadcast-related information may exist in various forms, such as,for example, EPGs (Electronic Program Guides) of DMB (Digital MultimediaBroadcasting) or ESGs (Electronic Service Guides) of DVB-H (DigitalVideo Broadcast-Handheld).

The broadcast receiving unit 111 may receive broadcast signals usingvarious broadcast systems. Broadcast signals and/or broadcast-relatedinformation received through the broadcast receiving unit 111 may bestored in the memory 160.

The Internet module 113 may refer to a module for access to theInternet. The Internet module 113 may be provided inside or outside theelectronic device 100.

The near-field communication module 114 refers to a module fornear-field communication. Near-field communication technologies mayinclude Bluetooth, RFID (Radio Frequency Identification), IrDA (InfraredData Association), UWB (Ultra Wideband), and ZigBee technologies.

The user input unit 120 is provided for a user's entry of audio or videosignals and may include a camera 121 and a microphone 122.

The camera 121 processes image frames including still images or videosas obtained by an image sensor in a video call mode or image capturingmode. The processed image frames may be displayed by the display unit151. The camera 121 may perform 2D or 3D image capturing or may beconfigured as one or a combination of 2D and 3D cameras.

The image frames processed by the camera 121 may be stored in the memory160 or may be transmitted to an outside device through the communicationunit 110. According to an embodiment, two or more cameras 121 may beincluded in the electronic device 100.

The microphone 122 receives external sound signals in a call mode,recording mode, or voice recognition mode and processes the receivedsignals as electrical voice data. The microphone 122 may perform variousnoise cancelling algorithms to remove noises created when receiving theexternal sound signals. A user may input various voice commands throughthe microphone 122 to the electronic device 100 to drive the electronicdevice 100 and to perform functions of the electronic device 100.

The output unit 150 may include a display unit 151 and a sound outputunit 152.

The display unit 151 displays information processed by the electronicdevice 100. For example, the display unit 151 displays a UI (UserInterface) or GUI (Graphic User Interface) associated with theelectronic device 100. The display unit 151 may be at least one of aliquid crystal display, a thin film transistor liquid crystal display,an organic light emitting diode display, a flexible display, and a 3Ddisplay. The display unit 151 may be configured in a transparent orlight transmissive type, which may be called a “transparent display”examples of which include transparent LCDs. The display unit 151 mayhave a light-transmissive rear structure in which a user may view anobject positioned behind the terminal body through an area occupied bythe display unit 151 in the terminal body.

According to an embodiment, two or more display units 151 may beincluded in the electronic device 100. For instance, the electronicdevice 100 may include a plurality of display units 151 that areintegrally or separately arranged on a surface of the electronic device100 or on respective different surfaces of the electronic device 100.

When the display unit 151 and a sensor sensing a touch (hereinafter,referred to as a “touch sensor”) are layered (this layered structure ishereinafter referred to as a “touch sensor”), the display unit 151 maybe used as an input device as well as an output device. The touch sensormay include, for example, a touch film, a touch sheet, or a touch pad.

The touch sensor may be configured to convert a change in pressure orcapacitance, which occurs at a certain area of the display unit 151,into an electrical input signal. The touch sensor may be configured todetect the pressure exerted during a touch as well as the position orarea of the touch.

Upon touch on the touch sensor, a corresponding signal is transferred toa touch controller. The touch controller processes the signal togenerate corresponding data and transmits the data to the control unit180. By doing so, the control unit 180 may recognize the area of thedisplay unit 151 where the touch occurred.

The sound output unit 152 may output audio data received from thecommunication unit 110 or stored in the memory 160. The sound outputunit 152 may output sound signals associated with functions (e.g., callsignal receipt sound, message receipt sound, etc.) performed by theelectronic device 100. The sound output unit 152 may include a receiver,a speaker, and a buzzer.

The memory 160 may store a program for operation of the control unit180, and may preliminarily store input/output data (for instance, phonebooks, messages, still images, videos, etc.). The memory 160 may storedata relating to vibrations and sounds having various patterns, whichare output when the touch screen is touched.

The memory 160 may include at least one storage medium of flash memorytypes, hard disk types, multimedia card micro types, card type memories(e.g., SD or XD memories), RAMS (Random Access Memories), SRAM (StaticRandom Access Memories), ROMs (Read-Only Memories), EEPROMs(Electrically Erasable Programmable Read-Only Memories), PROM(Programmable Read-Only Memories), magnetic memories, magnetic discs,and optical discs. The electronic device 100 may operate in associationwith a web storage performing a storage function of the memory 160 overthe Internet.

The interface unit 170 functions as a path between the electronic device100 and any external device connected to the electronic device 100. Theinterface unit 170 receives data or power from an external device andtransfers the data or power to each component of the electronic device100 or enables data to be transferred from the electronic device 100 tothe external device. For instance, the interface unit 170 may include awired/wireless headset port, an external recharger port, awired/wireless data port, a memory card port, a port connecting a devicehaving an identification module, an audio I/O (Input/Output) port, avideo I/O port, and an earphone port.

The control unit 180 controls the overall operation of the electronicdevice 100. For example, the control unit 180 performs control andprocesses associated with voice call, data communication, and videocall. The control unit 180 may include an image processing unit 182 forimage process. The image processing unit 182 is described below inrelevant parts in greater detail.

The power supply unit 190 receives internal or external power undercontrol of the control unit 180 and supplies the power to each componentfor operation of the component.

The embodiments described herein may be implemented in software orhardware or in a combination thereof, or in a recording medium readableby a computer or a similar device to the computer. When implemented inhardware, the embodiments may use at least one of ASICs (applicationspecific integrated circuits), DSPs (digital signal processors), DSPDs(digital signal processing devices), PLDs (programmable logic devices),FPGAs (field programmable gate arrays, processors, controllers,micro-controllers, microprocessors, and electrical units for performingfunctions. According to an embodiment, the embodiments may beimplemented by the control unit 180.

When implemented in software, some embodiments, such as procedures orfunctions, may entail a separate software module for enabling at leastone function or operation. Software codes may be implemented by asoftware application written in proper programming language. Thesoftware codes may be stored in the memory 160 and may be executed bythe control unit 180.

FIG. 2 is a view illustrating an example where a user inputs a gestureto an electronic device as shown in FIG. 1.

Referring to FIG. 2, the electronic device 100 may capture the gestureof the user U and may perform a proper function corresponding to thegesture.

The electronic device 100 may be any electronic device having thedisplay unit 151 that can display images. The electronic device 100 maybe a stationary terminal, such as a TV shown in FIG. 2, which is bulkyand thus placed in a fixed position, or may be a mobile terminal such asa cell phone. The electronic device 100 may include the camera 121 thatmay capture the gesture of the user U.

The camera 121 may be an optical electronic device that performs imagecapturing in a front direction of the electronic device 100. The camera121 may be a 2D camera for 2D image capturing and/or a 3D camera for 3Dimage capturing. Although in FIG. 2 one camera 121 is provided at a topcentral portion of the electronic device 100 for ease of description,the number, location, and type of the camera 121 may vary as necessary.

The control unit 180 may trace a user U having a control right whendiscovering the user U. The issue and trace of the control right may beperformed based on an image captured by the camera 121. For example, thecontrol unit 180 may analyze a captured image and continuously determinewhether there a specific user U exists, whether the specific user Uperforms a gesture necessary for obtaining the control right, andwhether the specific user U moves or not.

The control unit 180 may analyze a gesture of a user having the controlright based on a captured image. For example, when the user U makes apredetermined gesture but does not own the control right, no functionmay be conducted. However, when the user U has the control right, apredetermined function corresponding to the predetermined gesture may beconducted.

The gesture of the user U may include various operations using his/herbody. For example, the gesture may include the operation of the usersitting down, standing up, running, or even moving. Further, the gesturemay include operations using the user's head, foot, or hand H. Forconvenience of illustration, a gesture of using the hand H of the user Uis described below as an example. However, the embodiments of thepresent invention are not limited thereto.

According to an embodiment, analysis of a hand gesture may be conductedin the following ways.

First, the user's fingertips are detected, the number and shape of thefingertips are analyzed, and then converted into a gesture command.

The detection of the fingertips may be performed in two steps.

First, a step of detecting a hand area may be performed using a skintone of a human. A group of candidates for the hand area is designatedand contours of the candidates are extracted based on the human's skintone. Among the candidates, a candidate the contour of which has thesame number of points as a value in a predetermined range may beselected as the hand.

Secondly, as a step of determining the fingertips, the contour of thecandidate selected as the hand is run around and a curvature iscalculated based on inner products between adjacent points. Since thefingertips show sharp variation of their curvatures, when a change in acurvature of a fingertip exceeds a threshold value, the fingertip ischosen as a fingertip of the hand. The fingertips thusly extracted maybe converted into meaningful commands during gesture-command conversion.

According to an embodiment, it is often necessary with respect to agesture command for a synthesized virtual 3D image (3D object) to judgewhether a contact has occurred between the virtual 3D image and a user'sgesture. For example, it may be necessary, as is often case, whetherthere is a contact between an actual object and a virtual object tomanipulate the virtual object interposed in the actual object.

Whether the contact is present or not may be determined by variouscollision detection algorithms. For instance, a rectangle bounding boxmethod and a bounding sphere method may be adopted for such judgment.

The rectangle bounding box method compares areas of rectanglessurrounding a 2D object for collision detection. The rectangle boundingbox method has merits such as being less burden in calculation and easyto follow. The bounding sphere method determines whether there iscollision or not by comparing radii of spheres surrounding a 3D object.

For example, a depth camera may be used for manipulation of a real handand a virtual object. Depth information of the hand as obtained by thedepth camera is converted into a distance unit for a virtual world forpurposes of rendering of the virtual image, and collision with thevirtual object may be detected based on a coordinate.

Hereinafter, an exemplary environment in which the embodiments of thepresent invention are implemented is described. FIG. 3 is a view fordescribing an environment according to an embodiment of the presentinvention.

Referring to FIG. 3, a first user U1 and a second user U2 are positionedin a first place and a second place, respectively. The first user U1 maybe a person who hosts a teleconference and/or provides lectures to anumber of other people including the second user U2, and the second userU2 may be a person who attends the teleconference hosted by the firstuser U1.

A voice and/or motion of the first user U1 may be obtained and convertedinto video data and/or audio data by an electronic device 200 arrangedin the first place. Further, the video data and/or audio data may betransferred through a predetermined network (communication network) toanother electronic device 300 positioned in the second place. The firstelectronic device 300 may output the transferred video data and/or audiodata through an output unit in a visual or auditory manner. The firstelectronic device 200 and the first electronic device 300 each may bethe same or substantially the same as the electronic device 100described in connection with FIG. 1. However, according to anembodiment, each of the first electronic device 200 and the firstelectronic device 300 may include only some of the components of theelectronic device 100. According to an embodiment, the components of thefirst electronic device 200 may be different from the components of thefirst electronic device 300.

FIG. 3 illustrates an example where the first electronic device 200obtains and transfers the video data and/or audio data and the firstelectronic device 300 outputs the transferred video data and/or audiodata. According to an embodiment, the first electronic device 200 andthe first electronic device 300 may switch to each other in light offunctions and operations, or alternatively, each of the first electronicdevice 200 and the first electronic device 300 may perform the wholefunctions described above.

For example, the first user U1 may transfer his image and/or voicethrough the first electronic device 200 to the first electronic device300 and may receive and output an image and/or voice of the second userU2. Likewise, the first electronic device 300 may also perform the samefunctions and operations as the first electronic device 200.

Hereinafter, an operation of an electronic device according to anembodiment of the present invention is described. For purposes ofillustration, the operation is performed by an electronic device havingthe same configuration as the electronic device 100 described inconnection with FIG. 1. As used herein, a “first electronic device”refers to the first electronic device 200 shown in FIG. 3, which isarranged in the first place, and a “second electronic device” refers tothe first electronic device 300 shown in FIG. 3, which is arranged inthe second place. However, the embodiments of the present invention arenot limited thereto.

FIG. 4 is a flowchart for describing a method of operating an electronicdevice according to an embodiment of the present invention.

Referring to FIG. 4, the method of operating an electronic deviceincludes a step S100 of obtaining a first image including a interestingobject and at least one peripheral object, a step S110 of obtainingrelevance of the at least one peripheral object for the interestingobject based on at least one of attributes including a state, a motion,and a location of the interesting object, a step S120 of determining aspecific peripheral object according to the relevance, a step S130 ofreplacing an area of the whole first image, which is occupied by theremaining peripheral objects other than the specific peripheral objectand the interesting object, with another image and generating a secondimage, and a step S140 of transmitting the second image to the firstelectronic device 300. Hereinafter, each step is described in greaterdetail.

The first electronic device 200 may obtain a first image IM1 includingan interesting object IOB and at least one peripheral object POB (S100).For instance, the control unit 180 controls the camera 121 to obtain thefirst image IM1 including a plurality of objects including the firstuser U1. The first image IM1 may include a video obtained by capturingthe first user U1 and a periphery of the user U1.

The interesting object IOB may refer to an object determined to be mostimportant in the obtained first image IM1, and may include the firstuser U1.

The peripheral object POB may refer to the whole objects except for theinteresting object IOB in the first image IM1, and may include a desk Dand a board B arranged in the first place and a wall W partitioning thefirst place.

FIG. 5 is a view for describing a first image obtained according to anembodiment of the present invention. FIG. 5 illustrates the first imageIM1 including an interesting object IOB that is the first user U1 shownin FIG. 3 and peripheral objects POB1, POB2, and POB3 that are the deskD, the board B, and the wall W, respectively, as shown in FIG. 3.

For purposes of illustration, as shown in FIG. 5, the interesting objecthereinafter refers to the first user U1. However, the interesting objectis not limited to the first user U1, and according to an embodiment,other objects, such as the board B, may be set as the interestingobject. For instance, when the first user U1 intends to project datanecessary for a meeting onto the board B or to take notes while hostingthe meeting in an environment as shown in FIG. 3, the board B may be setas the interesting board.

The first electronic device 200 may obtain relevance of the at least oneperipheral object POB for the interesting object IOB based on at leastone of attributes including a state, a motion, and a location of theinteresting object IOB (S110).

The first electronic device 200 may obtain, through the control unit180, information on attributes of the interesting object IOB byanalyzing the obtained first image IM1 (S110). The control unit 180 mayanalyze the obtained first image using an image processing technology toobtain the attribute information on the interesting object U1. In thecase that the camera 121 is a 3D camera, the control unit 180 may obtainthe attribute information on the interesting object by analyzing depthinformation on the interesting object U1 obtained by the camera 121. Aspecific method of the first electronic device 200 obtaining theattribute information on the interesting object is described below.

The first electronic device 200 may obtain relevance of the at least oneperipheral object POB with respect to the interesting object IOB basedon the attribute information on the obtained interesting object (S110).The “relevance” refers to information on how much each peripheral objectPOB is related to the interesting object IOB. The relevance may bedesigned to have two options, such as “Related” and “Not-related”, ormay be designed to have more detailed, graded options. The relevance maybe defined to have relative values with respect to each peripheralobject rather than to have absolute values. For instance, the relevancemay be defined in such a manner that “the relevance of the firstperipheral object is higher than the relevance of the second peripheralobject.”

Referring to FIGS. 6 to 10, a method is specifically described thatanalyzes the obtained first image IM1 to obtain attribute information onthe interesting object IOB, thereby obtaining relevance of theperipheral object POB with respect to the interesting object IOB.

FIGS. 6 to 10 are views for describing a method of obtaining relevanceof a peripheral object for an interesting object based on an attributeof the interesting object according to an embodiment of the presentinvention.

The first electronic device 200 may obtain and analyze variousattributes of the interesting object IOB and obtain relevance.

First, the first electronic device 200 may obtain a location attributeof the interesting object IOB and obtain the relevance based on thelocation attribute.

For instance, as shown in FIGS. 6 and 7, the first electronic device 200may obtain a depth value of the interesting object IOB through thecamera 121 which is a 3D camera. When obtaining the location attributeof the interesting object IOB, the first electronic device 200 may alsoobtain depth values of other peripheral objects POB1 and POB2 includedin the first image IM1.

Based on the obtained depth values, the first electronic device 200 maycalculate a difference in distance between the interesting object IOBand each peripheral object POB1 or POB2. FIG. 6 illustrates an examplewhere a distance d1 between the interesting object IOB and the firstperipheral object POB1 is shorter than a distance d2 between theinteresting object IOB and the second peripheral object POB2, and FIG. 7illustrates an example where a distance d3 between the interestingobject IOB and the first peripheral object POB1 is longer than adistance d4 between the interesting object IOB and the second peripheralobject POB2.

The first electronic device 200 may allocate higher relevance to aperipheral object located closer to the interesting object IOB. Forinstance, in the example as shown in FIG. 6, the first electronic device200 may allocate higher relevance to the first object POB1 and lowerrelevance to the second object POB2, and in the example as shown in FIG.7, the first electronic device 200 may allocate higher relevance to thesecond object POB2.

Secondly, the first electronic device 200 may obtain a state attributeon the interesting object IOB and obtain the relevance based on thestate attribute.

For instance, as shown in FIGS. 8 and 9, the first electronic device 200may obtain information on a state of the interesting object IOB byanalyzing the first image IM1 obtained through the camera 121.

According to an embodiment, the state attribute of the interestingobject IOB may be defined by various methods.

For instance, the state attribute of the interesting object IOB may bedefined based on a physical contact state with other peripheral objectsPOB.

FIG. 8 illustrates a state where part (for example, the first user'shand) of the interesting object IOB contacts the first peripheral objectPOB1 (C1), and FIG. 9 illustrates a state where part of the interestingobject IOB contacts the second peripheral object POB2 (C2).

The first electronic device 200 may allocate higher relevance to aperipheral object contacting the interesting object IB. For example, inthe example as shown in FIG. 8, the first electronic device 200 mayassign higher relevance to the first peripheral object POB1 than thesecond peripheral object POB2, and in the example as shown in FIG. 9 thefirst electronic device 200 may allocate higher relevance to the secondperipheral object POB2 than the first peripheral object POB1.

According to an embodiment, the state attribute of the interestingobject IOB may be defined based on orientation of a specific part of theinteresting object IOB toward other peripheral objects POB.

FIG. 8 illustrates a state where part (for example, the first user'sarm) of the interesting object IOB extends in a direction (D1) towardthe first peripheral object POB1, and FIG. 9 illustrates a state wherepart of the interesting object IOB extends in a direction (D2) towardthe second peripheral object POB2.

The first electronic device 200 may allocate higher relevance to aperipheral object to which the interesting object IOB or at least partof the interesting object IOB is oriented. For example, in the exampleas shown in FIG. 8, the first electronic device 200 may allocate higherrelevance to the first peripheral object POB1 than the second peripheralobject POB2, and in the example as shown in FIG. 9 the first electronicdevice 200 may allocate higher relevance to the second peripheral objectPOB2 than the first peripheral object POB1.

According to an embodiment, in the case that the interesting object IOBis a human, such as the first user U1, the state attribute of theinteresting object IOB may be defined based on the direction of thefirst user's view.

For example, according to an embodiment, the first user's view isoriented toward the first peripheral object POB1, the first electronicdevice 200 may allocate higher relevance to the first peripheral objectPOB1 than the second peripheral object POB2. According to an embodiment,in the case that the first user's view is oriented toward the secondperipheral object POB2, the first electronic device 200 may allocatehigher relevance to the second peripheral object POB2.

Third, the first electronic device 200 may obtain a motion attribute onthe interesting object IOB and obtain the relevance based on the motionattribute.

For instance, the first electronic device 200 may obtain information ona motion of the interesting object IOB by analyzing the first image IM1obtained through the camera 121.

While obtaining the information on the motion of the interesting objectIOB, the first electronic device 200 may analyze directivity of themotion. The first electronic device 200 may analyze the directivity of amovement of the whole or part of the interesting object IOB.

FIG. 10 illustrates an example where at least part (for example, thefirst user's right arm RA) of the interesting object IOB moves and thefirst electronic device 200 may analyze a direction in which the part RAoff the interesting object IOB moves.

The first electronic device 200 may allocate higher relevance to aperipheral object that is located over a direction (motion direction) inwhich the interesting object IOB or at least part of the interestingobject IOB moves. For example, since in the example as shown in FIG. 10the second peripheral object POB2 is located over a direction (motiondirection) in which the right hand RA of the interesting object IOBmoves, the first electronic device 200 may allocate higher relevance tothe second peripheral object POB2 than the first peripheral object POB1.

Referring back to FIG. 4, the first electronic device 200 may select aspecific peripheral object according to the relevance (S120).

In step S120, the first electronic device 200 may select a peripheralobject having higher relevance between the peripheral objects POB1 andPOB2. For instance, in the example described in connection with FIGS. 6and 8, the first peripheral object POB1 may be selected as the specificobject, and in the example described in connection with FIGS. 7, 9, and10, the second peripheral object POB2 may be selected as the specificobject.

According to an embodiment, in step S120, the first electronic device200 may select one or two specific objects of the plurality ofperipheral objects POB1 and POB2, or according to an embodiment, thefirst electronic device 200 may select two or more specific objects.

For instance, when the first user U1 that is the interesting object IOBcontacts the first peripheral object POB1 as shown in FIG. 8 and thenthe first user U1 contacts the second peripheral object POB2 as shown inFIG. 9, the first and second peripheral objects may be both selected asthe specific objects.

According to an embodiment, when the first user's view is orientedtoward the second peripheral object POB2 and the first user's handcontacts the first peripheral object POB1, the first and secondperipheral objects POB1 and POB2 both may be selected as the specificperipheral objects.

According to an embodiment, when the first user's hand contacts thefirst peripheral object POB1 and the other hand of the first user U1 isoriented toward the second peripheral object POB2, the first and secondperipheral objects POB1 and POB2 may be both selected as the specificperipheral objects.

According to an embodiment, two or more peripheral objects may beselected as the specific peripheral objects based on variouscombinations.

Subsequently, the first electronic device 200 may replace areas occupiedby the other remaining peripheral objects than the specific peripheralobject and the interesting object IOB in the whole area of the firstimage IM1 with another image (S130).

FIGS. 11 and 12 are views each illustrating a second image in which anarea occupied by the remaining objects is replaced with another imageaccording to an embodiment of the present invention.

FIG. 11 illustrates an example where the first peripheral object POB1 isselected as a specific object. As described in connection with FIGS. 6and 8, in the case that the first peripheral object POB1 is selected asthe specific object, the interesting object IOB and the first peripheralobject POB1 are maintained, as is, in the first image IM1, but theremaining peripheral objects POB2 and POB3 are replaced by anotherimage, for example, a wavy sea image as shown in FIG. 11.

FIG. 12 illustrates an example where the second peripheral object POB2is selected as a specific object. As described in connection with FIGS.7, 9, and 10, in the case that the second peripheral object POB2 isselected as the specific object, the interesting object IOB and thesecond peripheral object POB2 are maintained, as is, in the first imageIM1, but the remaining peripheral objects POB1 and POB3 are replaced byanother image, for example, a wavy sea image as shown in FIG. 11.

The first electronic device 200 may transmit the second image generatedin steps S100 to S130 to the first electronic device 300 (S140).

By doing so, the first user U1 who hosts a meeting and/or provides adistance lecture through the first electronic device 200 may transmit animage including only his desired objects (for example, his own andspecific peripheral objects) among objects in the first place where thefirst user has the meeting to the first electronic device 300 used byanother user (for example, the second user). In particular, since thespecific peripheral objects desired to be displayed may be selectedwithout any complicated procedures, the first user may smoothly perform,e.g., a teleconference.

Although it has been described above that the relevance of theperipheral objects POB is obtained based on various attributes such aslocation and/or motion of the interesting object IOB included in theobtained first image IM1 and specific peripheral objects to be displayedtogether with the interesting object IOB are selected according to theobtained relevance, the embodiments of the present invention are notlimited thereto. According to an embodiment, the first user U1 may alsoselect the specific peripheral objects by directly selecting thespecific peripheral objects in the first image IM1. Hereinafter, amethod of selecting the specific objects is described.

FIGS. 13 and 14 are views for describing a method of selecting aspecific object according to an embodiment of the present invention.

As shown in FIGS. 13 and 14, the first electronic device 200 may displaythe obtained first image IM1 on the display unit 151 provided in thefirst electronic device 200 or connected to the first electronic device200 and may select the specific object according to the first user'sselective entry on the first image IM1.

FIG. 13 illustrates an example where the first peripheral object POB1 isselected as the specific object by the first user's touch input usinghis finger F, and FIG. 14 illustrates an example where the secondperipheral object POB2 is selected as the specific object by a touch onthe second peripheral object POB2.

Although in FIGS. 13 and 14, the first user's selective entry forselection of the specific object is a touch input, the selective entryfor selecting the specific object is not limited to the touch input andother input methods, such as entry using, for example, a mouse, akeyboard, a gesture, or a voice command, may be also used for theselective entry.

According to an embodiment, the first user's selective entry may beperformed on the obtained first image without performing the steps S110to S120 or after performing the steps S110 to S120.

When the first user's selective entry is performed after performing thesteps S110 and S120, the first image IM1 displayed on the display unit151 and viewed by the first user U1 may reflect information on thespecific peripheral object determined through the steps S110 and S120.Various methods may be used that reflect the information on the specificobject and displays the first image IM1.

For example, the selected specific object may be highlighted anddisplayed. According to an embodiment, the other remaining objects thanthe selected object may be blurred and displayed. For example, the firstelectronic device 200 may display the first image IM1 in such a way thatthe specific object selected through the steps S110 and S120 isdifferent in display property than the remaining peripheral objects, sothat information on the selected specific object may be reflected to thefirst image IM1.

According to an embodiment, another object, such as text and/or a symbolnotifying that the specific object has been selected, may be displayedat a periphery of the selected specific object so that the informationon the selected specific peripheral object may be reflected to the firstimage IM1.

Hereinafter, an operation of an electronic device according to anembodiment of the present invention is described. For purposes ofillustration, the operation is performed by an electronic device havingthe same configuration as the electronic device 100 described inconnection with FIG. 1. As used herein, a “first electronic device”refers to the first electronic device 200 shown in FIG. 3, which isarranged in the first place, and a “second electronic device” refers tothe first electronic device 300 shown in FIG. 3, which is arranged inthe second place. However, the embodiments of the present invention arenot limited thereto.

FIG. 15 is a flowchart for describing a method of operating anelectronic device according to an embodiment of the present invention.

Referring to FIG. 15, the method of operating an electronic deviceincludes a step S200 of obtaining a first image of an interesting image,a step S210 of determining a specific object based on a first motion ofthe interesting object, a step S220 of obtaining a virtual imagecorresponding to the specific object, a step S230 of determining adisplay state of the virtual image based on a second motion of theinteresting object, a step S240 of obtaining a final image including thevirtual image and at least part of the first image considering thedetermined display state, and a step S250 of transmitting the finalimage to the second electronic device. According to an embodiment, basedon various motions of the first object (for example, the first usershown in FIG. 3), an virtual image for various data (for example,presentation materials) that may be referred to for performing ateleconference may be synthesized with the image that is to betransmitted to another user (for example, the second user shown in FIG.3) and the synthesized image may be transmitted. Hereinafter, each stepis described in greater detail.

FIG. 16 is a view for describing a first place according to anembodiment of the present invention. Referring to FIG. 16, the firstplace includes a first object OB1 (U1) that hosts a teleconference, asecond object OB2 that is a desk, a third object OB3 that is a board,and a fourth object OB4 that is a document on the second object OB2. Forconvenience of description, the first object OB1 that is a first user U1is hereinafter referred to as an interesting object.

In the situation shown in FIG. 16, the first electronic device 200 mayobtain a first image for the interesting object OB1 (S200). The firstimage may include a video, and the first image may include at least oneof the second to fourth objects OB2, OB3, and OB4 as well as the imagefor the first object 031.

Subsequently, the interesting object OB1 may make a predetermined firstmotion. When obtaining the first motion, the first electronic device 200may determine all and/or at least some of the second to fourth objectsOB2, OB3, and OB4 as specific objects based on the first motion (S210).The predetermined first motion may vary and accordingly various methodsmay be used for determining the specific objects. The type of variousfirst motions and a method of determining the specific objects based onthe first motions are described below in greater detail.

The first electronic device 200 may obtain a virtual image for thespecific object determined in step S210 (S220).

The virtual image may be obtained by separating only an image for thespecific object from the first image.

According to an embodiment, the virtual image may be obtained from animage for the specific object obtained separately from the first image.For example, the first electronic device 200 may include two or morecameras 121. If the specific object is determined when the first imageis obtained by a first camera of the cameras, an image for the specificobject may be obtained by a second camera. The second camera may be ahigh-resolution camera designed to be able to obtain images havinghigher resolution than the first camera obtaining the first image.

The virtual image may be a 2D image for the specific object or a 3Dimage. In the case of obtaining the virtual image for the specificobject as a 3D image, the following additional operations may beseparately performed.

For instance, the specific object may be rotated by 360 degrees so thatthe interesting object (for example, the first user) may obtain a 3Dvirtual image for the selected specific object, and accordingly, thefirst electronic device 200 may obtain the 3D virtual image for thespecific object through the camera 121. The specific object may berotated by 360 degrees with respect to all of the three axes in a 3Dorthogonal coordinates system.

According to an embodiment, when obtaining a virtual image for theinteresting object, the 3D virtual image may be obtained using aplurality of images obtained based on images acquired by at least two ormore cameras.

The first electronic device 200 may separately store the obtainedvirtual image in the memory 160 so that the virtual image may be usedlater.

Subsequently, the first electronic device 200 may determine a displaystate of the virtual image based on the second motion (S230). Thedisplay state may include a location where the virtual image issynthesized, and size, orientation, a rotational state, and a markingstate of the virtual image. A method of determining the display state ofthe virtual image based on the second motion is specifically describedbelow.

When in step S230 the display state of the virtual image is determined,the first electronic device 200 may obtain a final image including thefirst image and the virtual image in consideration of the determineddisplay state (S240) and may transmit the final image to the firstelectronic device 300.

Hereinafter, referring to FIGS. 17 to 23, the type of first motions anda method of determining specific objects according to the first motionsare specifically described.

FIGS. 17 to 23 are views for describing the type of first motions and amethod of determining specific objects based on the first motionsaccording to an embodiment of the present invention.

Firstly, a first motion may include a grip for a specific object.

FIG. 17 illustrates an example where the interesting object OB1 gripsthe fourth object OB4. When the interesting object OB1 has a grip on thefourth object OB4, the first electronic device 200 determines the fourthobject OB4 as the specific object and obtains and separately stores avirtual image for the fourth object OB4.

However, the first electronic device 200 analyzes a gesture of theinteresting object OB1 to determine that the interesting object OB1makes a predetermined first motion. When as shown in FIG. 18 it isdetermined that there is no object associated with the grip even whenthe interesting object OB1 is analyzed to makes a gesture correspondingto the grip—for example, no object is located between the two hands suchas area R shown in FIG. 18), the first electronic device 200 disregardsthe gripping motion of the interesting object OB1.

For example, when obtaining the predetermined first motion, the firstelectronic device 200 may further analyze whether the predeterminedfirst motion is associated with another object, and then only when thefirst motion is associated with the other object, the first electronicdevice 200 may determine the associated object as the specific object.

Referring to FIG. 19A, the interesting object OB1 located in the firstplace releases the grip on the fourth object OB4 to determine thespecific object in step S210 and places the fourth object OB4 on thedesk OB2.

Referring to FIG. 19B, even when the fourth object OB4 is not actuallyincluded in an image obtained through the camera 121, a virtual imageVOB1 for the fourth object OB4 is synthesized and included in the finalimage FIM1 generated to be transmitted to the first electronic device300.

Secondly, the first motion may be a motion for designating an area inthe specific object—this motion may be simply referred to as an “areadesignating motion”.

FIG. 20 illustrates an example where the interesting object OB1 makes agesture of drawing a looped curve for an area R in the third object OB3.When the interesting object OB1 makes a motion for designating the areaR in the third object OB3, the first electronic device 200 determinesthe area R as the specific object and obtains and separately stores avirtual image for the area R.

When it is determined that no object is associated with the motion fordesignating the area even when making a gesture corresponding to thearea designating motion, the first electronic device 200 disregards thearea designating motion of the interesting object OB1.

For instance, in the case that the interesting object OB1 makes the areadesignating motion while a hand and/or a finger of the object OB1contacts the object or while the hand and/or finger is located within apredetermined distance of the object, the first electronic device 200may determine that the area designating motion is associated with theobject, and the first electronic device 200 may otherwise determine thatthe area designating motion is not associated with the object.

Referring to FIG. 21A, the interesting object OB1 located in the firstplace releases the area designating motion for the area R in the thirdobject OB3 for determining the specific object in step S210 and performsa teleconference. Referring to FIG. 21B, even though the area R in thethird object OB3 is not actually included in an image obtained throughthe camera 121, a virtual image VOB2 for the area R is synthesized andincluded in a final image FIM2 generated to be transmitted to the firstelectronic device 300.

Third, the first motion may be a tap on a specific object.

FIG. 22 illustrates an example where the interesting object OB1 makes atapping gesture on the fourth object OB4. When the interesting objectOB1 makes the tapping gesture on the object, the first electronic device200 determines the fourth object OB4 as the specific object, obtains andseparately stores a virtual image for the fourth object OB4. In the casethat it is determined that there is no object associated with the tapeven when it is analyzed that a gesture corresponding to the tap ismade, the first electronic device 200 disregards the tap of theinteresting object OB1. For instance, when the tapping gesture is madein the air, the first electronic device 200 may determine that there isno object associated with the tap and may disregard the tapping gesture.

Referring to FIG. 23A, the interesting object OB1 located in the firstplace releases the tapping motion for the fourth object OB4 fordetermining the specific object in step S210 and performs ateleconference. Referring to FIG. 23B, even when the fourth object OB4is not actually included in an image obtained through the camera 121, avirtual image VOB3 for the fourth object OB4 may be synthesized andincluded in a final image FIM3 generated to be transmitted to the firstelectronic device 300.

The virtual image for the specific object obtained by theabove-described method may be stored in the memory 160 separately froman image for the interesting object OB1 continuously obtained in realtime.

The separately stored virtual image may be used for performing stepsS230 and S240. Hereinafter, steps S230 and S240 are specificallydescribed.

After determining the specific object by the above-described method andobtaining the virtual image for the specific object, the firstelectronic device 200 determines a state for displaying the virtualimage to synthesize the virtual image with images continuously obtainedin real time through the camera 121.

First, the first electronic device 200 may determine a location wherethe virtual image is to be displayed.

The first electronic device 200 may determine the location where todisplay the virtual image by various methods.

For instance, the first electronic device 200 may analyze an imageobtained in real time and determine a location except for an areaoccupied by the interesting object OB1 as a location where to displaythe virtual image.

According to an embodiment, the first electronic device 200 may identifya specific sign (for example, a marker) from an image obtained throughthe camera 121 and may determine the location where to display thevirtual image based on the location of the specific sign. For instance,the interesting object OB1 may place a specific sign at a location whereto display the virtual image on the second object OB2, which is a desk.When the first electronic device 200, which previously stores an imagefor the specific sign, identifies the image for the sign, the firstelectronic device 200 may determine a location determined based on thesign as the location where to display the virtual image.

According to an embodiment, the first electronic device 200 maydetermine the location where to display the virtual image based on agesture input of the interesting object OB1.

According to an embodiment, the first electronic device 200 maydetermine the location where to display the virtual image based on adirection in which the interesting object OB1 views.

The first electronic device 200 may continuously determine the locationwhere to display the virtual image in real time. For example, the firstelectronic device 200 may once determine the location where to displaythe virtual image and may subsequently change the location where todisplay the virtual image rather than keeps displaying the virtual imageat the determined location. For example, as the location of theinteresting object OB1 varies in the obtained image, the location whereto display the virtual image may keep changing. In the obtained image,as the location of the marker changes, the location where to display thevirtual image may continue to change. As the gesture of the interestingobject OB1 and/or the direction in which the interesting object OB1views, the location where to display the virtual image may keepchanging.

The first electronic device 200 may determine a size of the virtualimage to be displayed.

The first electronic device 200 may determine the size of the virtualimage to be displayed by various methods.

For example, the first electronic device 200 may determine the size ofthe virtual image to be displayed in consideration of a size of a spaceexcept for an area occupied by the interesting object OB1 by analyzingan image obtained in real time.

For example, according to an embodiment, the first electronic device 200may determine the size of the virtual image to be displayed inconsideration of a size of text included in the virtual image so thatthe text may be displayed to have a size in which the text can beconveniently recognized by people.

For example, according to an embodiment, the first electronic device 200may determine the size of the virtual image to be displayed based on agesture input of the interesting object OB1.

The first electronic device 200 may continue to determine in real timethe size of the virtual image to be displayed. For example, the firstelectronic device 200 may once determine the size of the virtual imageto be displayed and may subsequently change the size of the virtualimage to be displayed rather than displays the virtual image based onthe determined size. For example, as the size of the area of theinteresting object OB1 varies in the obtained image, the size of thevirtual image to be displayed may keep changing.

FIGS. 24 and 25 are views for describing a method of changing a displaystate of a virtual image according to an embodiment of the presentinvention.

In the case that a final image FIM1 generated in step S240 is the sameas that shown in FIG. 24A, when the interesting object OB1 makes aspecific gesture G1 as shown in FIG. 24B, the final image FIM1 shown inFIG. 24A may change to a final image FIM4 shown in FIG. 24C. Forexample, the size of the virtual image VOB1 included in the final imageFIM1 may be enlarged according to the specific gesture G1 of theinteresting object OB1.

Similarly, when in the case of an example as shown in FIG. 25A, thespecific gesture G2 of the interesting object OB1 is sensed as shown inFIG. 25B, the first electronic device 200 may change a size of a virtualimage VOB2 included in a final image FIM2 as shown in FIG. 25A into asize of a virtual image VOB5 included in a final image FIM5 as shown inFIG. 25C. For example, the size of the virtual image may be reduced bythe specific gesture G2.

The first electronic device 200 may determine an orientation of thevirtual image to be displayed. For example, when the fourth object OB4is determined as the specific object in the example as shown in FIG. 22,the first electronic device 200 may obtain a virtual image for thefourth object OB4, wherein text included in the fourth object OB4 may bedisplayed upside down as shown in FIG. 23A. According to an embodiment,the first electronic device 200 may change the orientation of thevirtual image to be displayed as shown in FIG. 23B.

The first electronic device 200 may determine a degree of rotation ofthe virtual image to be displayed.

The first electronic device 200 may determine a location where todisplay the virtual image by various methods.

For instance, the first electronic device 200 may analyze an imageobtained in real time and may determine a rotational state of thevirtual, image according to a gesture of the interesting object OB1.

For example, according to an embodiment, the first electronic device 200may identify a specific sign (e.g., a marker) from an image obtainedthrough the camera 121 and may determine a rotational state of thevirtual image according to the rotation of the specific sign. Forexample, the first electronic device 200 may match a 3D coordinate axisof the virtual image to the specific sign and may rotate the virtualimage as the specific sign rotates.

According to an embodiment, the first electronic device 200 may continueto determine in real time the rotational state of the virtual image. Forinstance, the first electronic device 200 may once determine therotational state of the virtual image and subsequently change therotational state of the virtual image to be displayed rather thandisplays the virtual image based on the determined state.

FIGS. 26A, 26B, and 26C are views for describing a rotational state of avirtual image according to an embodiment of the present invention.

Referring to FIG. 26, in the case that a final image FIM3 generated instep S240 is as shown in FIG. 26A, when the final object OB1 makes aspecific gesture G3 as shown in FIG. 26B, the final image FIM3 shown inFIG. 26A may change to a final image FIM6 shown in FIG. 26C. Forexample, the virtual image VOB3 included in the final image FIM3 may berotated and displayed according to the specific gesture G3 of theinteresting object OB1.

The first electronic device 200 may determine a marking state of thevirtual image. For instance, the first electronic device 200 may putvarious markings, such as an underline, a highlight, or a square, on acontent such as a symbol and/or text included in the virtual image, andthe marked state may be reflected to the virtual image.

According to the embodiments of the present invention, user who hosts ateleconference and/or a distance lecture need not previously createdigitalized data of materials, such as audiovisual documentation,necessary for the conference and share the data with attendants.

Further, the embodiments may provide effects of being able to allow theobjects, such as handwritten data or shape of a prototype model, to beimmediately shared with other attendants while the conference is on theway. For example, a virtual image for an actual object may be instantlyobtained and synthesized with an image for the teleconference, and thesynthesized image may be transmitted to other users, thereby enablingdata necessary for the conference to be immediately shared with theusers in a convenience way.

Further, since data (e.g., objects) to be shared is stored as virtualimages and synthesized with images to be transmitted, a user hosting theconference may easily share desired objects with other users without aneffort to make to-be-shared objects oriented toward the camera (forexample, an effort for the host to proceed with the conference whileholding the to-be-shared data).

Further, a user may control the displayed location, size, orientation,and a display state, such as rotational state or marking state, of theinteresting object to be shared based on his gesture, so that theteleconference and/or distance lecture may be performed more smoothly.

In the methods of operating an electronic device according to theembodiments, each step is not necessary and according to an embodiment,the steps may be selectively included therein. The steps are notnecessary to perform in the order described above, and according to anembodiment, a later step may be performed earlier than an earlier step.

The steps in the methods of operating an electronic device may beperformed separately or in combination thereof. According to anembodiment, steps in a method may be performed in combination with stepsin another method.

The methods of operating an electronic device may be stored in acomputer readable medium in the form of codes or a program forperforming the methods.

The invention has been explained above with reference to exemplaryembodiments. It will be evident to those skilled in the art that variousmodifications may be made thereto without departing from the broaderspirit and scope of the invention. Further, although the invention hasbeen described in the context its implementation in particularenvironments and for particular applications, those skilled in the artwill recognize that the present invention's usefulness is not limitedthereto and that the invention can be beneficially utilized in anynumber of environments and implementations. The foregoing descriptionand drawings are, accordingly, to be regarded in an illustrative ratherthan a restrictive sense.

What is claimed is:
 1. A method of operating an electronic device, themethod comprising: obtaining a first image including an interestingobject and at least one peripheral object; obtaining relevance of the atleast one peripheral object with respect to the interesting object basedon at least one of attributes including a state, a motion, and alocation of the interesting object; and replacing an area of the wholearea of the first image, which is occupied by the interesting object andthe other remaining peripheral objects than a specific peripheral objectdetermined based on the relevance, with a second image.
 2. The method ofclaim 1, wherein obtaining the relevance of the at least one peripheralobject is performed in consideration of whether part of the interestingobject is connected to the peripheral object.
 3. The method of claim 1,wherein obtaining the relevance of the at least one peripheral object isperformed in consideration of a motion direction in which at least partof the interesting object moves.
 4. The method of claim 1, whereinobtaining the relevance of the at least one peripheral object isperformed in consideration of a distance between the interesting objectand the at least one peripheral object.
 5. The method of claim 1,wherein obtaining the relevance of the at least one peripheral object isperformed in further consideration of a user's input on the at least oneperipheral object.
 6. The method of claim 5, further comprising:displaying the first image; and receiving the user's input on thedisplayed first image.
 7. The method of claim 6, wherein displaying thefirst image includes making the interesting object and the firstperipheral object different in display property from the remainingperipheral objects.
 8. The method of claim 1, further comprisingtransmitting a third image generated by replacing the area occupied bythe remaining peripheral objects with the second image to a secondelectronic device.
 9. The method of claim 1, wherein the first imageincludes a moving image.
 10. A method of operating an electronic device,the method comprising: obtaining a first image of an interesting object;obtaining a virtual image corresponding to a specific object determinedbased on a first motion of the interesting object; obtaining a finalimage including the virtual image and at least part of the first imagein consideration of a display state of the virtual image determinedbased on a second object of the second motion; and transmitting thefinal image to a second electronic device.
 11. The method of claim 10,wherein the display state of the virtual image includes at least one ofa location where the virtual image is to be synthesized, a size, anorientation, a rotational state, and a marking state of the virtualimage.
 12. The method of claim 10, wherein obtaining the final imageincludes sensing the second motion in real time and changing the displaystate of the virtual image based on the sensed second motion.
 13. Themethod of claim 10, wherein obtaining the virtual image includes,determining whether the first motion is associated with the specificobject, and selecting the specific object only when the first motion isassociated with the specific object.
 14. The method of claim 10, whereinthe virtual image is a 2D image or a 3D image.
 15. An electronic devicecomprising: a camera; and a controller configured to obtain a firstimage including an interesting object and at least one peripheral objectthrough the camera, to obtain relevance of the at least one peripheralobject with respect to the interesting object based on at least one ofattributes including a state, a motion, and a location of theinteresting object, and to replace an area of the whole area of thefirst image, which is occupied by the interesting object and the otherremaining peripheral objects than a specific peripheral objectdetermined based on the relevance, with a second image.
 16. Theelectronic device of claim 15, wherein the controller is configured toobtain the relevance of the at least one peripheral object is performedin consideration of whether part of the interesting object is connectedto the peripheral object.
 17. The electronic device of claim 15, whereinthe controller is configured to obtain the relevance of the at least oneperipheral object is performed in consideration of a motion direction inwhich at least part of the interesting object moves.
 18. The electronicdevice of claim 15, wherein the controller is configured to obtain therelevance of the at least one peripheral object is performed inconsideration of a distance between the interesting object and the atleast one peripheral object.
 19. The electronic device of claim 15,further comprising: a communication unit, wherein the controller isconfigured to transmit a third image generated by replacing the areaoccupied by the remaining peripheral objects with the second image to asecond electronic device.
 20. An electronic device comprising: acommunication unit; a camera; and a controller configured to obtain afirst image of an interesting object through the camera, to obtain avirtual image corresponding to a specific object determined based on afirst motion of the interesting object, to obtain a final imageincluding the virtual image and at least part of the first image inconsideration of a display state of the virtual image determined basedon a second object of the second motion, and to transmit the final imageto a second electronic device through the communication unit.
 21. Theelectronic device of claim 20, wherein the display state of the virtualimage includes at least one of a location where the virtual image is tobe synthesized, a size, an orientation, a rotational state, and amarking state of the virtual image.
 22. The electronic device of claim20, wherein the controller is configured to sense the second motion inreal time and to change the display state of the virtual image based onthe sensed second motion.
 23. The electronic device of claim 20, whereinthe controller is configured to determine whether the first motion isassociated with the specific object, and to select the specific objectonly when the first motion is associated with the specific object.